Repository navigation
feat(providers): add opt-in local inference provider adapters - #48
Conversation
Register ollama, vllm, lmstudio, llamacpp and sglang as OpenAI-compatible providers only when their base URL variable is set (all unset by default). URLs are validated to an explicit /v1 endpoint, plain HTTP is limited to loopback and private addresses, and an optional provider key is used when configured; an explicitly configured local server may be called without one. No port scanning, discovery or downloads; catalog entries carry declared capabilities with provenance.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
multillm-proxy | b636129 | Oct 09 2026, 01:26 AM |
3e28b16
into
integrate/gateway-waves-1-4
Summary
Adds opt-in local inference providers:
ollama,vllm,lmstudio,llamacppandsglang. Each is an ordinary OpenAI-compatible upstream, active only when its base URL variable is set. With all five unset (the default), the provider list, catalog, model routes and every response are unchanged.ollamaOLLAMA_BASE_URLOLLAMA_API_KEYvllmVLLM_BASE_URLVLLM_API_KEYlmstudioLM_STUDIO_BASE_URLLM_STUDIO_API_KEYllamacppLLAMA_CPP_BASE_URLLLAMA_CPP_API_KEYsglangSGLANG_BASE_URLSGLANG_API_KEY<provider>:<model>on the existing managed/v1/chat/completions,/v1/responsesand/v1/modelsroutes. Only the OpenAI-compatible/v1API is used; Ollama's native/api/chatis not./v1./v1are rejected.localhostaddresses.Authorizationheader. Gateway authorization is still required. Other providers still refuse when their key is missing./v1/models.provider_catalog:<provider>provenance and never inferred from model names.Details are in
docs/local-inference.md.Coordinator edits in this PR
routes/provider_credentials.py:_provider_tokenconsultsAuthService.provider_requires_api_keybefore refusing a missing key. This lets an explicitly configured local server run keyless, while keyed providers keep refusing. It is the only change outside the builder's ownership row, and both cases are tested.worker/container-env.mjs: forwards the five base URL and five key variables to the Container.services/auth_service.pyis still 1,230 lines; this PR only adds small call sites there. Moving provider credential loading out of account authentication would be a reasonable follow-up.Tests
tests/test_local_inference.pycover:/v1normalization and URL policy;test_ledger_rows_hold_request_metadata_without_prompts_or_keys. It asserts that the ledger row doesn't contain the stringhi, and this run's random key prefix (mllm_yhihlRKN) happened to contain it. The file passes on rerun (11 passed), and this PR doesn't touch it.npm run test:worker: 1,460 passed.npm run test:integration: 7 passed.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.This is part of Wave 5. It is based on
integrate/gateway-waves-1-4, the Wave 4 integration branch. The README line comes with the Wave 5 integration PR.